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Abstract. We investigate the properties of a spin-orbit coupled quasi- two- 
dimensional Fermi gas with tunable s-wave interaction between the two spin species. 
By analyzing the two-body bound state, we find that the population of the excited 
states in the tightly-confined axial direction can be significant when the two-body 
binding energy becomes comparable or exceeds the axial confinement. Since the 
Rashba spin-orbit coupling that we study here tends to enhance the two-body binding 
energy, this effect can become prominent at unitarity or even on the BCS side of 
the Feshbach resonance. To study the impact of these excited modes along the 
third dimension, we adopt an effective two-dimensional Hamiltonian in the form of 
a two-channel model, where the dressed molecules in the closed channel consist of 
the conventional Feshbach molecules as well as the excited states occupation in the 
axial direction. With properly renormalizcd interactions between atoms and dressed 
molecules, we find that both the density distribution and the phase structure in the 
trap can be significantly modified near a wide Feshbach resonance. In particular, 
the stability region of the topological superfluid phase is increased. Our findings 
are helpful for the experimental search for the topological superfluid phase in ultra- 
cold Fermi gases, and have interesting implications for quasi-low-dimensional polarized 
Fermi gases in general. 
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1. Introduction 

Topological superfluid (TSF) phase is an unconventional superfluid state with superfluid 
gap in the bulk and topologically protected gapless modes at sample edges. As Majorana 
zero modes can emerge at the core of vortex excitations above the TSF state, these 
systems have attracted a considerable amount of attention recently [H El El IH E] . There 
have been various proposals for realizing the TSF state in condensed matter systems. 
In particular, J. Sau et al. suggested that such a state can be realized at the interface of 
semiconductor /superconductor heterostructures with spin-orbit coupling (SOC), s-wave 
pairing superfluidity and an external Zeeman field [5J. Notably, these key elements are 
also available in the context of quasi-two-dimensional (quasi-2D) ultra-cold Fermi gases 
near an s-wave Feshbach resonance. Importantly, with recent experimental realizations 
of SOC in ultra-cold atoms El EJ [TO], it seems plausible that Majorana fermions can 
be realized and manipulated in ultra-cold Fermi gases. 

For a polarized Fermi gas with tunable s-wave interaction, the phase diagram in the 
presence of SOC for both uniform and trapped cases have recently been systematically 
investigated HH H31 IT^l [T^l [THl HZl [THl HSl 1^01 I^Tl ■ In general, the competition 

between polarization, pairing superfluidity and SOC gives rise to rich phase structures. 
In three dimensions, two different topologically non-trivial superfluid phases with gapless 
excitations exist [TU [151 HB1 E3 US EE]- In two dimensions, a topological superfluid 
phase which supports Majorana zero mode at the core of vortex excitations can be 
stabilized [201 EI] • It has been demonstrated that the topological superfluid phase has 
large stability region near a wide Feshbach resonance, where the strong interaction also 
ensures the stability of the system against collisional losses and increases the critical 
temperature [221 [23] . This makes the quasi-2D polarized Fermi gas with SOC and 
strong interaction an ideal system for the realization of TSF state. Experimentally, 
quasi-2D Fermi gases are implemented by applying a strong confinement along the axial 
(z) direction and a weak harmonic trapping potential in the transverse (x-y) plane. 
Depending on the experimental design, it is possible to realize either a one-dimensional 
array [21] or a single layer [25] of quasi-2D Fermi gases. When the energy of the two- 
body bound state is small, the excited states in the axial direction are not significantly 
occupied at low temperatures. In this case, the axial degrees of freedom can be easily 
integrated out, leaving an effective two-dimensional (2D) Hamiltonian with renormalized 
atom-atom interactions [201 EZ1 EH] ■ However, with large two-body bound state energy, 
it has been shown that the occupation of axial excited state can be significant, and the 
degrees of freedom along the third dimension can lead to measurable many-body effects 
on the BEC side of the Feshbach resonance |29[ EH EI]- In this case, the many-body 
properties can be characterized by an effective two-channel model with renormalized 
interactions between atoms and dressed molecules, where the dressed molecules consist 
of Feshbach molecules and atoms in the axial excited states. 

In this work, we extend the effective two-channel model to describe quasi-2D Fermi 
gases with SOC. We first solve the two-body problem within quasi-2D confinement, and 
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find that the presence of SOC causes an increase of the excited modes population along 
the axial direction. This observation indicates that the third dimension would matter 
in a larger parameter region across the BCS-BEC crossover, and have to be taken 
into account to describe the quasi-2D, but truly three-dimensional (3D) system. We 
then adopt an effective two-channel model and determine the parameters by matching 
single- and two-body physics of the original Hamiltonian. We discuss the effects of 
dressed molecules on measurable many-body properties of the underlying system, and 
particularly, on the stability of the TSF phase. In general, we find that the dressed 
molecules can significantly modify the density profiles and the phase structure of the 
trapped gas near a Feshbach resonance. Typically, the phase boundaries between the 
TSF state and the conventional superfluid (SF) state are shifted in the trap. Under 
certain conditions, the inclusion of dressed molecules can qualitatively alter the in-trap 
phase structure and induce an additional TSF phase which is not present in a single- 
channel calculation. In either case, the stability region of the TSF phase is increased. 
Interestingly, we find that even a small closed-channel population can lead to significant 
changes in density profiles and/or phase structures. This suggests that these many-body 
effects may be probed experimentally near a wide Feshbach resonance. 

A qualitative understanding of the effects of dressed molecules can be achieved by 
considering the accommodation of population imbalance among various phases in the 
trap. For a polarized Fermi gas without SOC, typically, the competition between pairing 
and polarization can lead to the emergence of exotic phases, e.g., Fulde-Ferrel-Larkin- 
Ovchinnikov (FFLO) phase (32J [33] or the breached pairing (BP) phase [34] etc., in 
addition to the quantum phase transition between the SF and the normal states. In 
fact, as the conventional s-wave pairing superfluid state does not support population 
imbalance at zero temperature, normal phase or more exotic phases must exist in the 
trap to accommodate polarization. The situation is similar in the presence of SOC. 
Here, pairing parity is mixed, which leads to a small amount of population imbalance in 
the SF phase at zero temperature. However, this is overcome by the absence of normal 
state due to an SOC-induced pairing instability and the fact that dressed molecules 
in the two-channel Hamiltonian do not support population imbalance. We note that 
as the effects of SOC on the axial excited states have already been taken into account 
in the renormalization of the atom-molecule interactions, the dressed molecules in the 
effective two-channel model are merely structureless bosons formed due to short-range 
s-wave interactions. Hence, the lack of mechanism of the system to accommodate 
polarization otherwise leads to an enlarged stability region of the exotic TSF phase in a 
trap. As an interesting implication, we also note that the inclusion of dressed molecules 
in a polarized quasi-2D Fermi gas without SOC may also lead to measurable many- 
body effects, including a possible enhanced stability region of the exotic phases there. 
With recent experimental achievements in preparing and probing quasi-low-dimensional 
Fermi gases [351 H23 E3 EH], as well as the recent realization of SOC in ultra-cold Fermi 
gases 0, [TU] , our findings here should be helpful for future experiments. 

The paper is organized as follows: in Sec. 2, we consider a two-body problem in 
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quasi two dimensions in the presence of SOC and solve for the two-body bound state. 
We find that the population in the excited harmonic modes along the axial direction can 
become significant at unitarity of even on the BCS side of the resonance. To incorporate 
the effect of these excited modes, in Sec. 3 we write down an effective 2D Hamiltonian 
in a two-channel model form, and fix the parameters by matching single- and two-body 
physics. We then study typical phase structures of a trapped gas by using the effective 
two-channel model and discuss the effects of dressed molecules in Sec. 4. Finally, we 
summarize in Sec. 5. 



2. Two-body bound state in quasi-2D confinement 

We consider a two-component Fermi gas confined in a one-dimensional harmonic 
potential with trapping frequency lo z . In the presence of SOC, the system can be 
described by a conventional two-channel field theory around a Feshbach resonance [39, 



H = H + H soc + H hi + H mt . (1) 
The terms in the Hamiltonian are 



H = £/<^(-|5 + ^)^ 

+ J d 3 r^ + mfU 2 z z 2 + v^j 0, 

H u = g b J d 3 r (0V^t + H.C.) 

H iat = U b [ d 3 r^l^, (2) 



where ip a { r ) is the fermionic field operator with spin index o = 4>{f) is the 

bosonic molecular field operator, v h is the bare detuning, g b is the bare atom-molecule 
coupling constant, U b is the bare background scattering amplitude, and H.C. stands 
for Hermitian conjugate. Here, we assume a contact interaction between atoms. The 
bare scattering parameters are related to the physical ones (with subscript p) via the 
standard renormalization relations HH 



J (27r3)2eV U e * 

f 

U P = T 1 U b , g p = T g b , u p = u b + T-j-. (3) 

U c 

Here, e& = h 2 k 2 /(2rrif) is the dispersion relation for atoms with mass rrif and 3D 
momentum fc, and the integral is taken in three dimensions with an explicit 2D energy 
cutoff E c , so \J~ X = V r Ec/2 3//2 vr. The physical parameters can be obtained from the 
scattering measurements with 

- Anh 2 a hg _ Aiih?ii co W\a hg \ _ 

U P = -, 9 P =\ -, v v = i2 co (B - B ), (4) 
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where ctb g is the 3D background scattering length, W is the resonance width, /i co is the 
difference in magnetic moments of the closed and open channels, and B is the resonance 
position. 

The second term in the Hamiltonian Eq. represents the spin-orbit coupling 
(SOC). In the following discussion, we consider only the Rashba type SOC as 

■ — ih\ J d 3 rtjj^ (cr x d x + a y d y ) ip. (5) 

The parameter A is the SOC constant, ai =X)V are the Pauli matrices, and ip = (ip^,ip^) T . 
This SOC term couples particles with different spins such that spin is no longer a good 
quantum number. Instead, the single-particle Hamiltonian can be diagonalized in the 
helicity basis where each helix corresponds to particles with in-plane spin parallel or 
antiparallel to the in-plane momentum. 

To discuss the two-body bound state, we notice that the center-of-mass (CoM) 
degree of freedom along the axial direction is not affected by the interaction nor the 
SOC and can be separated from the relative coordinate under a harmonic potential. 
Thus, we can assume the CoM degree of freedom is in the ground harmonic mode along 
the ^-direction. In this CoM frame, the terms in Hamiltonian Eq. can be rewritten in 
a second quantized form by expanding the field operators ip G and (p m terms of harmonic 
oscillators along the ^-direction and plane waves in the x-y plane 

#0 = ^ (£m + e k ) cL,k,a C ™,M + ^ jyb + + Cq/2) ft| q frl,q> 
m,k,(T £,q 

H soc = A^ \(k x - ik y )c ] mM c mM + H.C.j , 

m,k 

Hbf = 9b Imni {b\,q C m, -k+q/2,4 C ™,k+q/2,t + H.C.j , 

m,n, £,k,q 

H in t =U b ^2 Imn C L,k+q/2,t C I,-k+q/2,; C «' ,-k'+q/2,4.Cm',k'+q/2,t- ( 6 ) 

m,n,k,m',n',k',q 

Here, we have used the z-direction trapping energy hw z as the energy unit, and 
its characteristic length a t = y/h/ {rrifUJ z ) as the length unit. The corresponding 
dimensionless parameters are defined as g b = gba^ 3 ^ 2 /(fko z ), Ub = UbaJ 3 /(fko z ), 
v h — Vb/{TujJ z )i an d A = Aa^ 1 / (hu z ). The bosonic field represents the molecular state 
with axial harmonic mode I and 2D transverse momentum q, which is also dimensionless 
in the unit of aJT 1 . The fermionic field c m ^ a represents atomic state with axial harmonic 
mode m and 2D momentum k, and is characterized with axial mode energy e m = m+1/2 
and plane wave energy e k = (A^+A^)/2 in the dimensionless form. The factors appearing 
in H^f and H\ nt are defined as overlap of harmonic oscillators 



2 1 / 4 / dz$ m (z)$Jz)$,(\/2z 
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77 i Ti 



\ dz$ m (z)<S> n (z)$ m/ (z)<S> n/ (z) = (7) 
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Here, $ n is the wavefunction of the n th harmonic oscillator 



(8) 



x) 

with H n (x) the n th Hermite polynomial. 

A general two-body state involving atoms and molecule can be written as the 
following ansatz 



in. 



(9) 

m,n,k <t,ct' 

where Yl'm n k indicates summation over mode (to, n) and transverse momentum k for 
k y > 0. The summation over spin runs over all four combinations of (a, a'). The 
coefficients /3^ q and ^ are determined by solving the Schrodinger's equation 

H\tye,<i — -E , ^,q|^ r )^q under the normalization relation. 

Under the quasi-2D condition, the z confinement is much greater than the Fermi 
energy E F and temperature k B T. In this case, the population of two-body states with 
nonzero CoM axial mode £ > is very limited, and the system's properties will be 
dominated by two-body physics in the £ = subspace. Next, we focus on two-body 
states with zero axial mode £ = 0, and obtain the equations determining the two-body 
bound state energy and the corresponding coefficients (3 and ?y's 

,2 -> -i 



S p( e <i) 



\n n I 

I lm,n, k,ql 



u n 



1 - Z^ q 

I 44 |2 

I lm,n, k,ql 



u p + e q /2 - E n 



A 2 |r? s 



m,n, k,q I 



E 2 

m,n, k,q 



k x 



-1/2 



X 2 <Jy( k x q y + /CyQ'a: 



£ 2 

m,n, k,q 



- AV 



+ (x <-> y) 



Vm,n, k,q 

4-t 

Vm,n, k,q 



(^7m,ri,k,q ^m,ri,k,q) ) 

(r; 

m,ri,k,q - ^,n,k,q) , 



where the functions are defined as 



2 

mnO 



m,n,k,q 



Z p( E ci) 

^7ra,n,k,q 
Vm,n, k,q 



£->. 

m,n,k 

9p~ — iy P + e q /2 - £ q ) , 



4A 2 A; 2 



4A 4 (fc^ - k y q x y 



m,n,k,q ~ - /- 2 . 2 2 n , 

^m,n,k,q Cm,n,k,q ^ m ,„,k,q A 9 J 

27mn0-^p 



T^m,n, k,q 
t2 



2iA 2 (-k x g y + fc^) s 

P , _ \2/r2 ^m,n,k,q 
°m,n,k,q ^ y 



(10) 

(11) 

,(12) 

(13) 
(14) 

(15) 

(16) 
(17) 
(18) 
(19) 
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Figure 1. Two-body bound state energy vs detuning for (a) 40 K and (b) 6 Li with 
transverse CoM momentum q = 0. (c) When plotted as functions of a t /a s , results for 
40 K and 6 Li are almost indistinguishable, showing universal behavior around unitarity. 
(d) The presence of SOC tends to enhance the two-body binding energy, which is the 
difference between bound state energy and open-channel threshold. 



with £ m ,n,k,q = E q — k 2 — 1 — m — n — g 2 /4. Here, we drop the subscript I = in i? q , 
/3 q and 77's to simplify notation. The dimensionless physical parameters are related to 
the bare ones via the same renormalization relations as in Eq. ([3]). The coefficients 
7mno = for m + n is odd, and 

(-l)(m-n)/2 ( m + n+l ^ 
7mn0 = ~ ^-F =fF (20) 



for m + n is even, where is the Euler Gamma function. 



The energy of two-body bound state is determined by solving Eq. (10), and the 



corresponding eigenstate can be extracted from Eqs. (11-14). As typical examples 



we focus on the s-wave wide Feshbach resonances around Bq = 202 G for 40 K and 
Bq = 834 G for 6 Li. The scattering parameters are taken to be the same as the cases 
without SOC, with W = 8 G, a bg = 174a B , fx co = 1.68/x B for 40 K @2J Hfl SI], and 
W = 300 G, abg = — 1405a£, fi co = 2/ie for 6 Li [45]. With a typical trapping frequency 
u z = 2tt x 62 kHz [21], the dimensionless physical parameters are then given by g p = 23 
(272), U p = 1.7 (-5.5) for 40 K ( 6 Li). We notice that in the presence of SOC, the open 
and closed channels involved in the Feshbach resonance will be altered since the spin 
is no longer a good quantum number. This modification can quantitatively change the 
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Figure 2. Population fraction of the two-body bound state in different axial harmonic 
levels for 40 K. Excited axial states are significantly populated around unitarity and on 
the BEC side of the resonance, as a direct consequence of the increasing two-body 
binding energy. The presence of SOC tends to enhance the binding energy, hence 
makes the population in excited states more prominent and may become notable even 
on the BCS side of the resonance. Results for 6 Li are similar as plotted as functions 
of a t /a s . 



resonance position as well as the scattering parameters. However, as long as we express 
the physical quantities in terms of the 3D scattering length a s , the results would remain 
valid due to the requirement of universality. 

We first consider two-body bound states with zero transverse CoM momentum 
q = 0. We show in Fig. [I] the eigenstate energy E q= o of two-body bound states for (a) 
40 K and (b) 6 Li. For both bound state is always present around the resonance 

point. This result is a combined effect of axial confinement and Rashba SOC, which both 
drive the system to an effective 2D model in the low-energy limit. For 40 K, the bound 
state energy saturates to a limiting value on the deep BEC side, as a direct consequence 
of its positive background scattering length [291 H5]- The bound state energy can also be 
illustrated as a function of the 3D scattering length, as shown in Fig. [l]^c). In this plot, 
results for 40 K and 6 Li are not distinguishable around the resonance point, indicating 
universal behavior at unitarity. 

In order to discuss the SOC effect, we show in Fig. []Jd) the binding energy of 
the bound state as the difference between the eigenstate energy and the open channel 
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threshold —\Eb(q = 0)| = E q=0 — E t ^. In the presence of SOC, the open channel 
consists of two colliding particles residing on the ground state of the lower helicity 
band. Thus, the threshold E t h = —A 2 + 1/2, where 1/2 denotes the zero-point energy of 
the relative degree of freedom along the confined direction. From Fig. [TJd), we notice 
that SOC tends to enhance the two-body binding energy throughout the entire BCS- 
BEC crossover region. This is understandable considering the fact that SOC increases 
the single particle density of states in the low-energy limit. 

In Fig. [2J we show the molecular fraction |/3| 2 and the population distribution 
in the axial harmonic levels within the two-body bound state for 40 K. The population 
distribution in the axial (m, n) mode is a combination of all four spin states P m .n — 
Yl'k \Vm'n,]i,q=o\ 2 ■ F rom these results, one can see clearly that many axial excited 
states are occupied. The population fraction still goes down as the energy of the modes 
goes up, but the convergence is poor and there are so many excited harmonics that the 
total population in the excited levels becomes significant at unitarity and eventually 
dominates on the BEC side of the resonance. This result is a direct consequence of 
the two-body binding energy approaching or exceeding the axial confinement, which 
can populate fermions to axial excited states. As SOC tends to enhance the two- 
body binding energy, this effect will be more eminent with stronger SOC, such that 
the population in excited harmonics can become dominant even on the BCS of the 
resonance. The results for 6 Li show similar behavior when plotted as functions of at/a s . 

Next, we discuss general two-body bound states with finite transverse CoM 
momentum. For cases with small q, we can expand the bound state energy as 
Eq pa E q= o + q 2 mf /(2m c g) with m e ff the effective mass of dimers. In the BCS limit with 
I E q= Q I <C A 2 , the molecular effective mass is m g — > 4m f, while in the BEC limit with 
|-Eq=o| ^> A 2 , m fj tends to the limiting value of 2m/. By tuning through the Feshbach 
resonance from the BCS to the BEC limit, the effective mass decreases monotonically 
as shown in Fig. |3^a). For a given scattering length, m e fj is larger for stronger SOC, as 
indicated in Fig. ^b). The effective mass also acquires universality where the results 
for 40 K and 6 Li are close around the resonance point. 

Up to now, we have discussed two-body bound state in a two-component Fermi 
gas with Rashba SOC confined in quasi two dimensions. We find that the axial excited 
harmonic levels will be significantly populated as the dimer's binding energy becomes 
comparable or exceeds the axial confinement energy. Compared to the case without 
SOC, this effect is more dramatic in the presence of SOC as the two-body binding 
energy is increased. As a direct consequence, the population of axial excited modes can 
be important even in the BCS regime provided that the SOC strength is large enough. In 
this case, these higher excited states must be taken into account for a correct description 
of the underlying system. For this purpose, next we present a 2D effective model which 
incorporates the effect of axial exited states into a so-called dressed molecule state. This 
model can describe the quasi-2D, but truly 3D Fermi gas in the low-energy limit. 
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Figure 3. Effective mass m e ff of the two-body bound state for 40 K. (a) In the presence 
of SOC, m c ff approaches 4m/ in the BCS limit, and decreases monotonically to the 
limiting value of 2m j in the BEC limit, (b) The effective mass becomes larger with 
stronger SOC intensity. Results for 6 Li are similar as functions of at/a s . 

3. Dressed molecule and effective 2D Hamiltonian 

In order to write down the correct effective 2D theory, we notice that the original quasi- 
2D Hamiltonian Eq. ^ contains three types of degree-of- freedom (DoF), including 
fermions in the axial ground state, fermions in axial excited states, and Feshbach 
molecules. These three types of DoF can be categorized into different length and energy 
scales. For fermions in the axial ground state, the corresponding length and energy 
scales are the inter-particle separation d and the Fermi energy Ep, respectively. For 
fermions in axial excited states, the corresponding length and energy scales are at and 
hw z . The Feshbach molecular state is related to the short-range details of the atom-atom 
interaction potential, which has length scale R e and energy scale V e . In ultra-cold Fermi 
gases with quasi-2D confinement, these three length and energy scales are usually well 
separated, satisfying d ^> a t ^> R e and Ep <C hco z <C V e . This observation allows us 
to write down an effective Hamiltonian to describe physics in the long-wavelength and 
low-energy limit. 

We combine the high-energy DoF including fermions in axial excited states and 
the Feshbach molecule to define a molecular state, which we call the dressed molecule 
since it can be viewed as the Feshbach molecule dressed by excited fermions [29J . Notice 
that the dressed molecule is structureless since all short-range details associated with 
excited fermions and the Feshbach molecule become irrelevant in the low-energy regime. 
With this dressed molecule and the fermions in the axial ground state, we write down 
an effective 2D Hamiltonian in the form of a two-channel model (also in dimensionless 
form with length unit a t and energy unit hui z ) 

k,cr q 

+ A' ^2 - ik y )al^a Kl + H.C. 

k 



Topological superfluid in a quasi-two-dimensional polarized Fermi gas 



11 



+ a b E ( rf q a k+q/2,t a -k+q/2,; + H.C.) 
k,q 

+ V b 22 a k+q/2,t a -k+q/2,4 a - k '+q/ 2 ^ ak '+q/ 2 't- ( 21 ) 
k,k',q 

Here, a^fa^) are fermionic creation (annihilation) operator in two dimensions, ek = 
k 2 /2 is the corresponding dispersion relation, and d denote the dressed molecular 
operators, and the Rashba SOC is characterized by the effective coupling constant A'. 
The three bare scattering parameters 5b, c*b, and Vb can be linked to physical ones via 
a 2D renormalization analogous toEq. § 

' (4^)2^ + 1' + K' 



^ = 0"% o p = fi- 1 a 6 , <5 p = 5 6 + ft^. (22) 

v r 



C 



The parameters in the effective Hamiltonian are determined by the following 
considerations. First, we require the effective model to give the correct open channel 
threshold, which represents the energy of two particles when they are far apart. In other 
words, the effective theory needs to reproduce the single-particle dispersion in the low- 
energy limit. Second, when the system is far away from resonance where the population 
of both the Feshbach molecule and excited fermions are negligible, the quasi-2D Fermi 
gas can be well described by a 2D system with all fermions staying in the axial ground 
state. This observation indicates that the background interaction Vb in the effective 2D 
theory should be related to the 3D background interaction Ub by integrating out the 
harmonic ground state wavefunction along the axial direction. 

By tuning through the Feshbach resonance, we further require the effective 2D 
model to reproduce the correct two-body physics. In fact, since the possibility for three 
and more particles come to a close range is exponentially small in dilute atomic gases, 
the low-energy physics is dominated by two-body processes. For the quasi-2D system 
around the BCS-BEC crossover, the ground state is always a two-body bound state 
with axial CoM mode I = and transverse CoM momentum q = 0. Therefore, the 
effective 2D theory should give the same two-body binding energy \E b (i — 0, q = 0)| 
as the original Hamiltonian. Besides, since the dressed molecule is a phenomenological 
combination of the Feshbach molecule and axial excited fermions, we also require the 
population of dressed molecule to match the population of Feshbach molecule plus that 
of the dimers formed with axial excited fermions. 

To study the two-body problem within the effective 2D model, we take the ansatz 
wavefunction as 



i $ ) q = (K+E 'E^ ct ' 

\ k <r,cr' 



U k+q/2,<7 -k+q/2 



v |0>, (23) 



where indicates summation over 2D momentum with k y > 0, and the summation 
over spins runs over all four combinations of (a, a'). We focus on the case with q = 
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a,/a s a,la s a,la s 



Figure 4. Parameters in the effective 2D Hamiltonian as functions of at/a s . The 
energy independent background scattering amplitude V p for different SOC constants 
are: 3.134 (A=0), 3.147 (A=l), 3.186 (A=2), 3.341 (A=4) for 40 K, and -1.801 (A=0), 
-1.828 (A=l), -1.914 (A=2), -2.155 (A=4) for 6 Li. 



and substitute the wavefunction into the Schrodinger's equation H e g\Q) q=0 = E b \Q) q=0 , 
which leads to 

„2 -| -1 

= Vfrra p (E b ), (24) 



p S n — Eh 



where the function a p is defined as 



d 2 k 

(2tt) 5 /2 

7r + 2tan _1 



+ 



b+2E b 



-b- 



^/-b(4E b +b) J ^ \n(-E b ) 



(25) 



8ny/27r y/-b(AE b + b) 

and b = 4(A') 2 is used to simplify notation. 

With these results, we can fix the parameters in the effective 2D Hamiltonian by 
matching single- and two-body physics as discussed above 

A' = A, (26) 
{Up 1 - C p ) , (27) 



v- 1 

p 



a, 



E h - 



o~ p (E b ) 



dV(E b )/dE b 
1 



2irdV(E b )/dE b 



o~ P {E b ) 
U~ — C p 

Q- p (E b ) 

Up 1 - c p 



(28) 
(29) 
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Here, the parameters are defined as 

C p =S p {Ef + 1/2)- a p {Ef), (30) 

V(E b ) = [l/Uf - S p (E b + 1/2) + a p (E b )] , (31) 

Uf = Up -g 2 p /(u p -E b ), (32) 

with E x b ni the two-body binding energy in quasi two dimensions for v p — > oo. Notice that 
the eigenenergy of the two-body bound state as determined by Eq. (10) is shifted from 
the binding energy by the zero-point energy along the strongly confined axial direction. 

Equations (26 29), together with the renormalization relation Eq. (22), fix the 
parameters in the effective 2D Hamiltonian (21 ) as functions of two-body binding energy 
E b , which is tuned via the physical detuning v p through the Feshbach resonance. In 
Fig. [4j we plot the parameters 5 P and a p for 40 K and 6 Li across resonance for various 
SOC strengths, using the same 3D parameters as in Fig. [TJ These effective parameters 
behave qualitatively differently for 40 K and 6 Li, mainly due to the difference in sign 
of their individual background interaction. We also show the effective interaction 
V p eS = V p — atp/(8 p — E b ), as it will be used for the mean-field calculation discussed 
below. 

Up to now, we have introduced an effective 2D Hamiltonian by matching single- and 
two-body physics with the original quasi-2D system. This effective theory is derived by 
grouping the high-energy DoF of axial excited fermions and Feshbach molecules to define 
a dressed molecular state, while keeping the low-energy axial ground fermions to catch 
the correct low-energy physics of the corresponding many-body system. To justify the 
validity of this effective theory, we notice that in the language of T-matrix, the matching 
conditions of two-body binding energy and molecular fraction correspond to the pole 
of two-body T-matrix and the first derivative of T -1 around the pole, respectively. 
Thus, we conclude that as the fermionic chemical potential 2/x is not far away from the 
bound state energy — \E b \, the T-matrix of the original quasi-2D Hamiltonian can be 
well approximated by the effective 2D model 

T(x) « T eS (x), as \x\ = |2/x + \E b \\ < 1. (33) 

Considering the fact that 2fi+\E b \ is of the same order as the Fermi energy Ep through 
the BCS-BEC crossover, and that the Fermi energy is proportional to the 2D number 
density Ep oc n 2 D, we conclude that the effective theory is approximately valid as the 
diluteness and the quasi-2D conditions hold for the underlying Fermi gas. 



4. Many-body calculations with the effective two-channel model 

With the bare parameters of the effective two-channel model fixed, we may now proceed 
with the many-body calculations. Here, we consider only the possibility of BCS 
superfluid and focus on the zero CoM momentum pairing state with q = 0. The 
dimensionless many-body Hamiltonian can be written as 

H eS - [PfN^ - /^iVj, = ^ ( e k - AO a k,a a k,<7 + (S b - 2/x)dJdo 

k,cr 
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Figure 5. Density distribution for spin- up (bold, red) and spin-down (thin, black) 
atoms in the single-channel (dotted) and the effective two-channel (solid) models, with 
parameters: (a) a t /a s ~ 0.34, A ~ 0.14, u) x /u z = uj x /uj z = 10~ 3 , N ~ f0 4 , P ~ 0.50, 

2N b /N ~ 13%; (b) a t /a s 0.40, A - 0.22, ui x /u z = u x /lo z = 10~ 3 , N ~ f0 4 , 

P ~ 0.27, 2Nb/N ~ 5%. For the effective two-channel calculation, the percentage of 
atoms in the closed channel 2N^/N — 2 J d?rrib(r)/N, where the density distribution 
of the closed channel atoms 2n,b are shown with the dash-dotted line (green). The 
first-order phase boundaries in (a) between the SF and TSF phases are indicated by 
the abrupt jump in the density distributions. The vertical dashed lines in (b) indicate 
the second-order phase boundaries between the SF and TSF phases in the two-channel 
model. 



k 

+ a b (4°k,t°-ia + H - C -) +V b^2 

a k,t a -k,4, a -k',4- a k',t' 



(34) 



where the bare parameters are fixed by the renormalization relation Eq. (22), together 
with Eqs. (26 29). Assuming a slow- varying harmonic potential in the x-y plane, 



we take the local density approximation (LDA) in the transverse direction such that 
/if = + h, Hi = //(f) — h, //(f) = /i(0) — V(f), where V(f) is the dimensionless 
external trapping potential, /i(f) is the local chemical potential, h is the chemical 
potential imbalance. To be consistent with the two-body calculations, we have taken 
the unit of energy to be hu z , and the unit of length along the zth (i = x,y) direction 
to be Ri = at0J z /ui. The dimensionless radial harmonic potential is then V(f) = r 2 /2, 
with f 2 = Y.i r i/ R i- 

Following Ref. |20j, the zero-temperature thermodynamic potential can be 
evaluated on the mean-field level 
IAI 2 . . . 

(35) 



n 



yes 
p 



E 



k,sJ 



k,s=± 



where the order parameter for the two-channel model A = a p (do) + V p J2k ( a -Kl a Kt)^ 
and V p eS = V P — a 2 / (6 P — E\,). The dispersion relations for quasi-particles in the presence 
of SOC are given as 



E, 



k,± 



H + \ 2 k 2 + |A| 2 + /i 2 ±2E , 



(36) 
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Figure 6. Density distribution for different interaction strengths with fixed SOC 



and polarization: (a-c) A 
cases, lo x /uj z — uj x /u> z = 

a t /a s 0.36, 2N b /N 

2N b /N ~ 20%; (d) a t /a s r 
a t /a s ~ 0, 2N b /N ~ 10%. 



< 0.14, P ~ 0.20; (d-f) A - 0.22, P ~ 0.27. In all 
10~ 3 , N ~ 10 4 , P ~ 0.20, with other parameters: (a) 
6%; (b) a t /a s ~ 0.04, 2N b /N ~ 13%; (c) a t /a s ~ 0.34, 
- -1, 2N b /N ~ 1%; (e) a t /a s ~ -0.5, 2N b /N ~ 4%; (f) 
The bold red (thin black) dotted curves are the density 
distribution of spin-up (spin-down) atoms in the single-channel model, and the solid 
curves are the density distribution in the effective two-channel model. The distributions 
for atoms in the closed channel are shown with the dash-dotted lines (green). In (a- 
c), the phase boundary between the SF and the TSF phases are first-order for both 
models. While in (d)(e), the vertical lines indicate the second-order phase boundaries 
between the SF and TSF phases in the two-channel model. In (f), as the TSF phase 
shows up in both models, we indicate the second-order phase boundary between the 
SF and TSF phases for the two-channel (single-channel) model using a vertical dashed 
line in the upper (lower) plane. 



where E = ^Jh 2 (££ + |A| 2 ) + A 2 ££A; 2 and £k = e k _ /•*(?)• Due to the competition 
between pairing and polarization, one needs to find the global minimum of the 
thermodynamic potential to avoid getting meta-stable or unstable states. 

The local chemical potentials can be determined self-consistently from the 
dimensionless number equations 

N = J rf 2 f[n t (f)+n 4 (f)], (37) 
P=lj d 2 f[n t (f)-n 4 (f))], (38) 

where h a = —dft/dfi a , N = Nco x co y /co^, with N the total particle number in the trap. 
Solving these equations while minimizing the local thermodynamic potential, we get 
density distributions of the gas in a typical quasi-2D trapping potential. The TSF 
phase appears in the trap where h > ■sj /i 2 (f) + |A(f)| 2 [HI [20] . Finally, we note that the 
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Figure 7. Density distribution for different polarization at resonance at/a s = 0. In 
all cases, A ~ 0.14, lo x /ui z = l>j x /uj z = 10~ 3 , N ~ 10 4 , with the polarizations and the 
closed-channel populations: (a) P ~ 0.10, 2N b /N ~ 13%; (b) P ~ 0.42, 2N b /N ~ 8%; 
(c) P ~ 0.82, 2N b /N ~ 2%. The bold red (thin black) dotted curves are the density 
distribution of spin-up (spin-down) atoms in the single-channel model, and the solid 
curves are the density distribution in the effective two-channel model. The distributions 
for atoms in the closed channel are shown with the dash-dotted lines (green). Only 
first-order phase boundaries appear here for both models. 



population of the dressed molecules are given as [IB] 

n 6 (r) = (4d ) = |A(f)| 2 

We now calculate the typical density distributions of a polarized quasi-2D Fermi 
gas with SOC. To see the effect of dressed molecules in the closed channel, we compare 
results from the effective two-channel model with those of a single-channel model [20J . 
In the context of quasi-2D gases, the single-channel model can be understood as an 
approximation in which the population of all excited levels in the axial direction are 
neglected. One may then integrate out the lowest axial mode and relate the two-body 
binding energy \Eb\ appearing in the renormalization condition in the single-channel 
model to the 3D scattering length a s via \E b \ « 0.9157T -1 exp ( \/27r at/a s ) Jtw z [26 | [27| 128] . 

In Fig. [5] we demonstrate the typical density distributions calculated from the two 
models. From these results, we find that the dressed molecules affect properties of the 
trapped gas in two different ways. First, the density distribution can be significantly 
modified by the inclusion of dressed molecules [c.f. Fig. [51(a)]. Second, in the presence 
of dressed molecules, the in-trap phase structure can be different. This is manifested in 
Fig. [5|a), where the first-order phase boundaries between the conventional superfluid 
phase (SF) at the center and the topological superfluid phase (TSF) at the edge are 
shifted. Notably, on the BCS side of the Feshbach resonance, the dressed molecules 
can qualitatively alter the in-trap phase structure by inducing an additional TSF phase 
which is absent in a single-channel calculation [c.f. Fig. [51(b)]. 

In Fig. [6j we systematically map out the density distributions for fixed total particle 
number and polarization while the interaction strength is tuned. Apparently, the 
difference in density distribution becomes more dramatic as the interaction strength is 
tuned toward the BEC side. This also applies to the shift in the phase boundary between 
the SF and TSF phases [Fig. [6] (a-c)]. On the other hand, the stability region of the 



U P (S P - 2/x(r)) 



a. 



(39) 
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TSF phase induced by the dressed molecules increases toward the BEC side [Fig. [6[d-f )] . 
The TSF phase in the single-channel model appears near unitarity, after which the two 
models differ only by a shift in the phase boundaries [Fig. |6^f)]. Importantly, in either 
case, the inclusion of dressed molecules results in significant changes to the many-body 
properties of the system that may be probed experimentally. 

The effects of dressed molecules on the density distribution in quasi-2D Fermi gases 
have been studied before for an unpolarized case without SOC [301 EI]- In that case, 
the modification of the density distribution is only significant in the deep BEC region. 
In the current case, however, the dressed molecules can have measurable effects near 
unitarity or even on the BCS side of the resonance. This is a combined effect of SOC and 
population imbalance. First, as discussed in the two-body calculation, SOC causes an 
increase of two-body binding energy, which can significantly populate the excited axial 
harmonic states and consequently enhance the effects of dressed molecules. Second, it 
is also related to the mechanism by which the population imbalance is accommodated 
in the system. Generally speaking, in a trapped Fermi gas, the population imbalance is 
accounted for by the normal phase toward the trap edge or by exotic superfluid phases 
that supports polarization, e.g. the FFLO phase [321 E2], the BP phase [34J, or the 
TSF phase in the presence of SOC [HJ [12]. The dressed molecules, on the other hand, 
do not support polarization. Therefore, when the dressed molecules are present, the 
in-trap density distribution must be modified to accommodate the total polarization 
fixed a priori. A particularly interesting observation here is that exotic phases may 
be induced in the trap to account for polarization even in the presence of very small 
dressed molecule populations [Fig. |6](d-e)]. We expect that this picture should hold for 
a quasi-2D polarized Fermi gas without SOC. 

Finally, we plot in Fig. [7] density distributions for various polarizations at resonance. 
As polarization increases, the population of the dressed molecules decreases. This can 
be understood in terms of the competition between polarization and closed-channel 
population, as larger polarization requires more population in the open channel. The 
results suggest that at resonance, the effects of dressed molecules are stronger at 
intermediate polarization. 

5. Summary 

In this work, we study a quasi-two-dimensional Fermi gas confined in a strong 
axial harmonic potential with Rashba spin-orbit coupling and population imbalance. 
We analyze the two-body bound state in such a system, and find that the axial 
excited harmonic levels can be significantly populated as the binding energy becomes 
comparable or exceeds the axial confinement. The presence of Rashba spin-orbit 
coupling increases the density of states in the low-energy limit, hence causes an 
enhancement of two-body binding energy, and eventually excites more particles to the 
high-lying axial harmonic modes. This observation implies that the degrees of freedom 
along the third dimension would be important for a satisfactory description of the quasi- 
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two-dimensional, but truly three-dimensional system. 

In order to incorporate the effects of these axial excited modes, we introduce 
a dressed molecule degree of freedom and construct an effective two-dimensional 
Hamiltonian in the form of a two-channel model. The parameters in this effective 
model is fixed by matching single- and two-body physics with the original Hamiltonian. 
Specifically, we require the effective Hamiltonian to give the correct single-particle 
dispersion, background scattering amplitude, two-body binding energy, and fraction 
of Feshbach molecules plus dimers formed with axial excited fermions. These matching 
conditions are equivalent to fixing the pole of two-body T-matrix and the first derivative 
of T _1 around the pole. Therefore, the effective model can mimic the low-energy physics 
as the original Hamiltonian provided that the chemical potential is not too far away from 
one half of the two-body binding energy. This condition usually holds for ultra-cold 
dilute Fermi gases in quasi-2D confinement. 

We then investigate many-body properties of a quasi-2D Fermi gas using this 
effective Hamiltonian. We conclude that the inclusion of dressed molecules are crucial 
for the investigation of the underlying system. Specifically, we find that the stability 
region of the topological superfluid phase is increased as a result of the occupation of 
axial excited modes. We then systematically map out the in-trap phase structures and 
density distributions throughout the whole BCS-BEC crossover region, and discuss the 
appropriate parameter region that the effects of the dressed molecules may be observed. 
Our findings are helpful for the experimental search for the topological superfluid phase 
in ultra-cold Fermi gases, and have interesting implications for quasi-low-dimensional 
polarized Fermi gases in general. 
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